Analysis of support vector machines

We compare L1 and L2 soft margin support vector machines from the standpoint of positive definiteness, the number of support vectors, and uniqueness and degeneracy of solutions. Since the Hessian matrix of L2 SVMs is positive definite, the number of support vectors for L2 SVMs is larger than or equal to the number of L1 SVMs. For L1 SVMs, if there are plural irreducible sets of support vectors, the solution of the dual problem is non-unique although the primal problem is unique. Similar to L1 SVMs, degenerate solutions, in which all the data are classified into one class, occur for L2 SVMs. INTRODUCTION As opposed to L2 soft margin support vector machines (L2 SVMs), L1 soft margin support vector machines (L1 SVMs) are widely used for pattern classification and function approximation. And much effort has been done to clarify the properties of L1 SVMs [1, 2, 3, 4]. Pontil and Verri [1] clarified dependence of the L1 SVM solutions on the margin parameter C. Rifkin, Pontil, and Verri [2] showed degeneracy of L1 SVM solutions, in which any data are classified into one class. Fernández [3] also proved the existence of degeneracy without mentioning it. Burges [4] discussed non-uniqueness of L1 SVM primal solutions, and uniqueness of L2 SVM solutions. But except for [4], little comparison has been made between L1 and L2 SVMs [5]. In this paper, we compare L1 SVMs with L2 SVMs from the standpoint of positive definiteness, the number of support vectors, and uniqueness and degeneracy of solutions. Since the Hessian matrix of L2 SVMs is positive definite, the solutions are unique. For the L1 SVMs, we introduce the concept of irreducible set of support vectors and show that if there are plural irreducible sets, the dual solutions are non-unique. Finally, we show that L2 SVMs have degenerate solutions similar to L1 SVMs. In the following, first we summarize L1 and L2 SVMs and discuss the Hessian matrices for L1 and L2 SVMs. Then we discuss non-uniqueness of L1 SVM dual solutions. Finally, we prove the existence of degeneracy for L2 SVMs. SOFT MARGIN SUPPORT VECTOR MACHINES In soft margin support vector machines, we consider the linear decision function D(x) = wg(x) + b (1) in the feature space, where w is the weight vector, g(x) is the mapping function that maps the m-dimensional input x into the l-dimensional feature space, and b is a scalar. We determine the decision function so that the classification error for the training data and unknown data is minimized. This can be achieved by minimizing 1 2 ‖w‖2 + C M ∑